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Key findings 



This study used data from the fall 2014 semester at Wisconsin Virtual School to 
examine whether patterns of student engagement in online courses (the amount of 
time a student was logged in to the online course each week and how this varied over 
time) were associated with course outcomes (the percentage of possible points earned 
in the course and the percentage of course activities completed). Key findings include: 
Student enrollments in online courses followed one of six engagement patterns, 
with average engagement ranging from 1.5 hours to 6 or more hours per week. 
Most students (77 percent) steadily engaged in their online course for 1.5 or 2.5 
hours per week. 

Students who engaged in their online course for at least 1.5 hours per week 
typically earned a high enough percentage of possible points to pass the course. 
Students who engaged in their online course for two or more hours per week had 
better course outcomes than students who engaged for fewer than two hours per 
week. 
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Summary 


Student enrollment in online courses has increased in the past 15 years and continues 
to grow (Watson, Pape, Murin, Gemin, & Vashaw, 2014)- However, little is known about 
students’ education experiences or online course outcomes. These are areas of particular 
interest to the Midwest Virtual Education Research Alliance, whose goal is to understand 
how to support student success in online courses. Members of the alliance partnered with 
Regional Educational Laboratory Midwest to develop and conduct this study on how stu- 
dents engage in online learning and how student engagement patterns are associated with 
online course outcomes. Findings from this study may help inform policymakers, state 
and local education agencies, and online learning providers as they seek ways to support 
student success in online courses. 

This study analyzed learning management system data and student information system data 
for all core, elective, and Advanced Placement online high school course enrollments during 
the fall 2014 semester. The data were collected by Wisconsin Virtual School, a statedevel 
online learning program that partnered with 194 Wisconsin districts to serve 5,511 student 
enrollments in 256 supplemental online courses during the 2014/15 school year. 

Analyses looked for student engagement patterns in online courses and the percentage of 
student enrollments that followed each pattern; differences among student engagement 
groups (groups of student enrollments that followed a given pattern) in course type taken, 
gender, or grade level; and associations between student engagement in online learning 
and online course outcomes. Engagement refers to behavioral engagement and was defined 
as the amount of time a student was logged in to the online course each week. Course 
outcomes were measured by the percentage of possible points earned in the course (which 
students’ home schools use to assign a letter grade based on the local grading scale) and 
the percentage of course activities completed. 

Analyses revealed six engagement patterns among student enrollments in Wisconsin Virtual 
School online courses. Each pattern represents an average of the total time a student spent 
logged in to the online course each week, 1 as well as how that amount varied across the 
semester (for example, increasing, decreasing, or remaining steady). The six student engage- 
ment patterns and the percentage of student enrollments in each were as follows: 

• Initial 1.5 hours with decrease (8 percent). Engagement of approximately 1.5 
hours per week at the beginning of the semester that drops off to near 0 hours 
midway through the semester. 

• Steady 1.5 hours (39 percent). Steady engagement of approximately 1.5 hours per 
week, with a slight increase toward the end of the semester. 

• Initial 2 hours with spike (4 percent). Engagement of approximately 2 hours per 
week at the beginning of the semester that increases steadily after the midpoint of 
the semester to nearly 12 hours per week in the final week. 

• Steady 2.5 hours (38 percent). Steady engagement of approximately 2.5 hours per 
week across the semester. 

• Steady 4+ hours (8 percent). Consistent engagement of approximately 4 hours or 
more per week across the semester. 

• Variable 6+ hours (2 percent). High but variable engagement ranging from 
approximately 6 hours per week at the beginning, middle, and end of the semester 
and peaking to near 10 hours between those points (during weeks 3 and 17). 


Students who engaged in their online course for at least 1.5 hours per week (all groups 
but the initial 1.5 hours with decrease group) typically earned a high enough percentage 
of possible points to pass the course even though they varied in total time logged in each 
week initially and across the semester. However, the steady 1.5 hours group performed sig- 
nificantly worse than the initial 2 hours with spike, steady 2.5 hours, steady 4+ hours, 
and variable 6+ hours groups in percentage of possible points earned in the course and 
percentage of course activities completed. 

Although this study is not designed to determine whether the relationship between student 
engagement patterns and course outcomes is causal, educators may want to use the find- 
ings to identify students who need additional support to succeed — for example, the initial 
1.5 hours with decrease and steady 1.5 hours engagement groups, which had poorer course 
outcomes. In addition, online learning programs across the country may be able to use the 
findings as a framework for investigating the data available in their learning management 
systems and student information systems. 
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Why this study? 


Student enrollment in online courses has increased in the past 15 years and continues 
to grow (Watson et al., 2014). Districts and public high schools enroll students in online 
courses for a variety of reasons, such as to provide students with opportunities to take 
courses not otherwise available to them or to take collegedevel courses, as well as to per- 
sonalize student learning or to provide students with an alternative learning environment 
(Means, Bakia, & Murphy, 2014; Queen & Lewis, 2011). In addition, five states (Alabama, 
Arkansas, Florida, Michigan, and Virginia) require that students take at least one online or 
blended online and face-to-face course to graduate high school (Watson et al., 2014). Spe- 
cific requirements vary across states, with one state (Alabama) requiring that the online 
course be in a core subject area and all but one state (Virginia) requiring that the online 
course be credit bearing. In all cases the requirement is intended to broaden students’ edu- 
cational experiences to prepare them for college and the workplace. Definitions of online 
courses and other key terms used in this report are provided in box 1. 

However, little is known about students’ educational experiences or online course out- 
comes. Student failure to complete online courses is a substantial problem for some online 
learning programs (Angelino, Williams, & Natvig, 2007). And although providing stu- 
dents the opportunity to personalize their learning is often stated as a potential benefit 
of online learning (International Association for K-12 Online Learning, 2011a), little is 
known about whether or how students take advantage of the opportunity. A recent study 
conducted in collaboration with Regional Educational Laboratory Midwest’s Virtual Edu- 
cation Research Alliance found that two of the greatest challenges for online teachers at 
Wisconsin Virtual School — a state-level online learning program that partners with dis- 
tricts in the state to offer supplemental online courses for middle and high school students 
(see box 2 for a description of Wisconsin Virtual School) — were keeping students engaged 
throughout the course and getting students to complete the course (Zweig, Stafford, Clem- 
ents, & Pazzaglia, 2015). 


This study 
examines how 
students engage 
in online learning 
and how student 
engagement 
patterns are 
associated with 
online course 
outcomes 


The current study examines how students engage in online learning and how student 
engagement patterns are associated with online course outcomes. Such information may 
help inform policymakers, state and local education agencies, and online course providers 
in developing or refining online programs and promoting student success in online courses. 
Although this study is not designed to determine whether the relationship between 
student engagement patterns and course outcomes is causal, educators may want to use the 
findings to identify students who need additional support to succeed. In addition, the study 
provides a framework for how other online learning programs can use the data available in 
their learning management systems and student information systems to examine student 
engagement and online course outcomes. 


Types of student engagement 

In response to concerns that schools are not adequately preparing students for success in 
college and beyond, some researchers and policymakers have placed a spotlight on how 
factors such as student engagement may play an important role in academic success (Far- 
rington et al., 2012; Lawson & Lawson, 2013; Rosen, Glennie, Dalton, Lennon, & Bozick, 
2010). Although research on student engagement includes many definitions and concep- 
tualizations, one recent literature review identified three types of engagement — affective, 
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Box 1. Key terms 


Advanced Placement. A program offered by the College Board that gives high school students 
the opportunity to take college-level courses for college credit. 

Core course. A course required for high school graduation. In Wisconsin core courses are 
English language arts (four credits), social studies (three credits), math (three credits), science 
(three credits), and physical education (1.5 credits; Wisconsin Statutes and Annotations, 
2014). 

Course outcome. The percentage of possible points earned in the course or the percentage of 
course activities completed. 

Credit recovery course, A course that allows students to obtain credit for classes they have 
failed. 

Elective course. A course other than an Advanced Placement, core, or credit recovery course. 

Learning management system. The technology platform through which students access online 
courses. It generally includes software for creating and editing course content, communication 
tools, assessment tools, and other features for managing the course (International Associa- 
tion for K-12 Online Learning, 2011b). 

Online course. Any course offered over the Internet (International Association for K-12 Online 
Learning, 2011b). 

Online learning. Education in which instruction and content are delivered primarily over the 
Internet (that is, electronically). The term does not include print correspondence courses; 
courses delivered by broadcast television or radio, CD, or videocassette; or standalone edu- 
cational software programs that do not have a substantial Internet-based instructional compo- 
nent (International Association for K-12 Online Learning, 2011b). 

Online learning program. An organized offering of courses delivered primarily over the Internet 
(International Association for K-12 Online Learning, 2011b). 

Online teacher. A staff member of the online learning program who teaches students in online 
courses. 

Onsite facilitator. A staff member at a brick-and-mortar school responsible for monitoring and 
supporting students in the school who are taking online courses. The staff member filling this 
role can range from a classroom teacher to a guidance counselor to an aide. 

Student engagement, Researchers have identified several types of engagement. This study 
focuses on behavioral engagement and defines student engagement as the amount of time 
that a student is logged in to the online course. 

Student engagement group. A group of students who follow a given student engagement 
pattern. 

Student engagement pattern. A pattern of student engagement measured and modeled over 
time, such as throughout the duration of a course. 

Student information system. A system for storing and managing student data, including 
contact information; demographic characteristics such as gender, grade level, age, race/ 
ethnicity, special education status, and English learner student status; and previous academic 
performance. 
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Box 2. Wisconsin Virtual School 


Wisconsin Virtual School is a state-level online learning program that partners with districts 
throughout Wisconsin to offer 256 supplemental online courses for middle and high school stu- 
dents. In operation since 2000, Wisconsin Virtual School served 5,511 student enrollments in 
194 Wisconsin districts during the 2014/15 school year. The online learning program employs 
54 Wisconsin-certified online teachers, who are expected to be actively involved in planning 
instruction, assessing learning, and communicating with students. Wisconsin Virtual School 
also requires that partnering districts have an onsite facilitator (called a “local education guide”) 
to monitor progress and provide face-to-face support for students enrolled in online courses. 

The online courses are asynchronous, which means that students and teachers typical- 
ly do not log in to the online course at the same time. Students are expected to log in to 
the learning management system daily to check email and course announcements and are 
encouraged to communicate with their teacher at least once a week. While logged in, students 
review course content (including interactive audio and video clips), complete practice activi- 
ties, post on discussion boards, attend real-time online chat sessions with the teacher and 
other students, and complete assignments for each unit sequentially before taking quizzes or 
exams. All coursework must be submitted through the learning management system, but some 
courses include assignments that require reading, writing, or other work that occurs outside 
the learning management system. As such, the act of logging in to an online course is concep- 
tually similar to spending time in a class in a brick-and-mortar school and possibly some time 
doing homework, depending on the nature of the particular course and assignment. 

Wisconsin Virtual School provides certificates of completion for all students who complete 
95-100 percent of the possible course points and reports the percentage of points earned to 
students’ schools. The home district then determines whether 95-100 percent course activity 
completion with a passing grade, which is determined locally, provides an earned credit or 
advancement. 


cognitive, and behavioral — and found that all three are positively associated with student 
success in traditional face-to-face schooling (Lawson & Lawson, 2013). The current study 
focuses on behavioral engagement, which includes the amount of time students spend on 
coursework. For online courses, engagement can be partially measured by the amount of 
time a student is logged in to the online course each week, which is collected by learning 
management systems (the online platforms that deliver online courses). 

Student engagement in online courses 

Although research and recommendations exist for promoting student engagement in 
face-to-face school settings, it is unclear whether those recommendations translate to the 
online environment, because online settings involve unique considerations and challenges. 
The challenges include physical distance, lack of interaction, and psychological distance 
(Moore, 1993; Murphy & Rodriguez-Manzanares, 2008; Rovai, 2002). Furthermore, inves- 
tigating engagement in online learning may be particularly important in light of research 
suggesting that student attrition and course failure are problems for some online programs 
(Carr, 2000; Freidhoff, 2015; Freidhoff, DeBruler, & Kennedy, 2014; Roblyer, 2006; Roblyer 
& Davis, 2008; Rovai & Wighting, 2005; Simpson, 2004). Understanding whether and how 
students’ engagement in online learning is associated with course outcomes may provide a 
foundation for future research on how to support student engagement and success. 


It is unclear 
whether 
research and 
recommendations 
for promoting 
student 
engagement in 
face-to-face school 
settings translate 
to the online 
environment, 
because 
online settings 
involve unique 
considerations 
and challenges, 
including physical 
distance, lack of 
interaction, and 
psychological 
distance 
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This study builds on recent research using learning management system data and student 
information system data to identify factors that predict student success in online courses. 
Hung, Hsu, and Rice (2012) found that students with higher engagement (as measured by 
the total number of learning management system activities across the span of the course, 
including frequency of login, frequency of modules accessed, frequency of clicks, and fre- 
quency of discussion board posts) tended to have higher final grades. Liu and Cavanaugh 
(2011) found that the number of times a student logged in to the learning management 
system throughout the course and the total number of minutes a student was logged in 
throughout the course were associated with students’ final scores in two online biology 
courses offered through a Midwestern state virtual school. They also found that the total 
number of minutes that a student was logged in throughout the course was the stron- 
gest predictor of final scores in a model that accounted for student demographics and the 
number of teacher comments throughout the course. Because these studies were correla- 
tional in nature, it is not possible to determine the direction or extent of causality between 
engagement and course outcomes. It is possible that factors such as motivation or academic 
ability cause both student engagement and course outcomes. 

The current study builds on these earlier studies by identifying distinct patterns of student 
engagement within online courses over time and exploring whether the patterns are asso- 
ciated with online course outcomes. 


The total amount of 
time a student was 
logged in to the 
online course each 
week was used to 
investigate student 
engagement 
patterns in 
online courses 


What the study examined 


This study used data collected by Wisconsin Virtual School’s learning management system 
and student information system (see box 2 for a description of Wisconsin Virtual School). 
In particular, the total amount of time a student was logged in to the online course each 
week was used to investigate student engagement patterns in online courses. 

Three research questions guided the study: 

• How many student engagement patterns are evident in online courses, and what 
percentage of student enrollments follow each pattern? 

• Do student engagement groups differ in course type taken, gender, or grade level? 

• How are student engagement patterns associated with the percentage of possible 
points earned in the course and the percentage of course activities completed? 

The research questions were addressed by first identifying patterns of student engagement 
with online courses and then examining the relationship between the patterns and course 
outcomes (see box 3 for a summary of the study’s data and methodology and appendix A 
for more detail). 


What the study found 


Student enrollments followed one of six engagement patterns in Wisconsin Virtual School 
online courses. In five of the six engagement groups students typically earned a high 
enough percentage of possible points to pass their course, and students who engaged for 
two or more hours per week had better course outcomes than students who engaged for 
fewer than two hours per week. 
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Box 3. Data and methods 


Data 

Wisconsin Virtual School provided student information system data and learning management 
system data. The student information system data included course characteristics, student 
demographic characteristics, and course outcomes (percentage of possible points earned in 
the course and percentage of course activities completed). The learning management system 
data included the amount of time each week that each student was logged in from August 1, 
2014, to February 28, 2015. The two data files were merged using Wisconsin Virtual School’s 
student IDs to link the files. 

This study included 1,512 student enrollments from 109 Wisconsin Virtual School 
Advanced Placement (15 percent of student enrollments), core (36 percent), and elective 
(49 percent) high school courses from the fall 2014 semester. The study did not analyze 
credit recovery courses because those courses have a shortened timeline. The sample includ- 
ed 1,179 unique students, 170 of whom took more than one online course. Sixty percent of 
student enrollments were female students, and 40 percent were male students. The majority 
of student enrollments were in grades 9-12: 11 percent were in grade 9, 25 percent were in 
grade 10, 28 percent were in grade 11, and 32 percent were in grade 12. Three percent of 
student enrollments were in grades 5-8. 

Methodology 

To answer the research question on student engagement patterns, the study team used group- 
based trajectory modeling, which first identifies patterns of behavior across time and then 
assigns individuals to pattern-based groups. To answer the research question on differences 
in course type taken, gender, and grade level across student engagement patterns, descriptive 
statistics of those characteristics were calculated for each engagement group, and tests of 
statistical significance were used to compare the characteristics across groups. To answer 
the research question on the association between student engagement and course outcomes, 
regression analysis that accounts for how students are similar within online courses was used; 
it compared the outcomes of students displaying the most common engagement pattern with 
students displaying each of the other engagement patterns, while statistically adjusting for 
student and course characteristics. 


Student enrollments in online courses followed one of six engagement patterns 

The amount of time a student was logged in to the online course each week was best 
represented by a model with six student engagement patterns (figure 1). The two most 
common patterns each accounted for almost 40 percent of student enrollments; the least 
common pattern accounted for just 2 percent. The six patterns and the percentage of 
student enrollments in each are: 

• Initial 1.5 hours with decrease ( 8 percent). Engagement of approximately 1.5 
hours per week at the beginning of the semester that drops off to near 0 hours 
midway through the semester. 

• Steady 1.5 hours (39 percent). Steady engagement of approximately 1.5 hours per 
week, with a slight increase toward the end of the semester. 

• Initial 2 hours with spike (4 percent). Engagement of approximately 2 hours per 
week at the beginning of the semester that increases steadily after the midpoint of 
the semester to nearly 12 hours per week in the final week. 
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• Steady 2.5 hours (38 percent). Steady engagement of approximately 2.5 hours per 
week across the semester. 

• Steady 4+ hours (8 percent). Consistent engagement of approximately 4 hours or 
more per week across the semester. 

• Variable 6+ hours (2 percent). High but variable engagement ranging from 
approximately 6 hours per week at the beginning, middle, and end of the semester 
and peaking to near 10 hours between those points (during weeks 3 and 17). 

Student enrollments in the six engagement groups differed in course type taken, gender 
composition, and grade-level composition 

Several statistically significant differences in characteristics emerged across student engage- 
ment groups: 

• The initial 2 hours with spike, steady 2.5 hours, steady 4+ hours, and variable 6+ 
hours groups each had a higher percentage of student enrollments in Advanced 
Placement courses (19-31 percent) than the initial 1.5 hours with decrease 
(7 percent) and steady 1.5 hours groups (9 percent; table 1). 

• The initial 2 hours with spike group had a higher percentage of male student enroll- 
ments (57 percent) than the steady 1.5 hours, steady 2.5 hours, steady 4+ hours, and 
variable 6+ hours groups, which had close to the sample average (40 percent). 

• The initial 1.5 hours with decrease and variable 6+ hours groups had a higher 
percentage of student enrollments in grade 9 (20 percent and 17 percent) than the 
other groups (9-11 percent). 


Figure 1. Student enrollments in Wisconsin Virtual School online courses followed 
one of six engagement patterns, fall 2014 


Hours per week spent logged in to the online course 



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 

Weeks in online course 


Note: Analysis is based on 1,512 student enrollments. The maximum number of data points for each student 
enrollment is 21 weeks, though some students finished their course in fewer than 21 weeks. This approach 
uses all available data for each week. Values in parentheses are the percentage of the sample that followed 
each engagement pattern; percentages do not sum to 100 because of rounding. The student engagement 
patterns depicted were stylized using the predicted values for each week. See figure B1 in appendix B for con- 
fidence bands around the predicted patterns and estimates of population percentages for each pattern. 

Source: Authors' analysis of Wisconsin Virtual School data (fall 2014 semester). 


The two most 
common student 
engagement 
patterns each 
accounted for 
almost 40 percent 
of student 
enrollments; the 
least common 
pattern accounted 
for just 2 percent 
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Table 1. Engagement groups differed in course type taken, gender, and grade level, 
fall 2014 

Percent of student enrollments 

Characteristic 

Initial 
1.5 hours 
with 

decrease 
(n = 122) 

Steady 
1.5 hours 
(n = 592) 

Initial 

2 hours 
with 
spike 
(n = 68) 

Steady 
2.5 hours 
(n = 572) 

Steady 

4+ hours 
(n = 122) 

Variable 
6+ hours 
(n = 36) 

Full 

sample 

Course type taken (n = 1,512) 

Advanced Placement 

7 

9 

25 

19 

31 

25 

15 

Core 

48 

34 

34 

35 

32 

44 

36 

Elective 

46 

57 

41 

46 

37 

31 

49 

Gender ( n = l,501 a ) 

Female 

57 

61 

43 

60 

60 

66 

60 

Male 

43 

39 

57 

40 

40 

34 

40 

Grade level ( n = l,501 a ) 

5-8 

4 

3 

1 

2 

6 

3 

3 

9 

20 

10 

9 

11 

9 

17 

11 

10 

18 

27 

25 

25 

27 

26 

25 

11 

20 

30 

38 

28 

24 

34 

28 

12 

38 

30 

26 

35 

34 

20 

32 


a. 11 student enrollments (less than 1 percent of the total) were missing information on gender or grade level. 
Percentages exclude missing values. 

Note: Percentages may not sum to 100 because of rounding 

Source: Authors' analysis of Wisconsin Virtual School data (fall 2014 semester). 


Table B1 in appendix B provides additional descriptive statistics, and table B2 in appen- 
dix B provides tests of statistical significance. 

Most students in five of the six engagement groups earned a high enough percentage of possible 
points to pass their online course 

Most students in the five engagement groups with average engagement of at least 1.5 hours per 
week (which includes all groups except the initial 1.5 hours with decrease group) earned a high 
enough percentage of possible points to pass their online course, though average total time 
logged in each week and the change in that average over time varied across groups (figure 2). 

Students with low but steady engagement in their online course had better outcomes than students 
with low initial engagement that diminished throughout the course 

The steady 1.5 hours engagement group had better course outcomes than the initial 1.5 
hours with decrease group after student and course characteristics were controlled for 
(table 2). The average percentage of possible points earned in the online course was 35 per- 
centage points higher for the steady 1.5 hours group than for the initial 1.5 hours group, 
and the average percentage of course activities completed was 63 percentage points higher 
for the steady 1.5 hours group than for the initial 1.5 hours with decrease group. 


The steady 1.5 
hours a week 
engagement group 
had better course 
outcomes than the 
initial 1.5 hours 
with decrease 
group after 
student and course 
characteristics 
were controlled 
for; the average 
percentage of 
course activities 
completed was 
63 percentage 
points higher for 
the steady 1.5 
hours group than 
for the initial 
1.5 hours with 
decrease group 
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Figure 2. Most students in five of the six engagement groups earned a high enough 
percentage of possible points to pass their online course, though their engagement 
patterns differed, fall 2014 


■ Percentage of possible points earned in the course ■ Percentage of course activities completed 
Initial 1.5 hours with decrease (n = 122) 

Steady 1.5 hours (n = 592) 

Initial 2 hours with spike (n = 68) 

Steady 2.5 hours (n = 572) 

Steady 4+ hours (n = 122) 

Variable 6+ hours (n = 36) 

0 20 40 60 80 100 

Average student outcome 

Note: Data were statistically adjusted for student demographic characteristics and course type taken (Ad- 
vanced Placement, core, or elective). Wisconsin Virtual School provides the percentage of possible points 
earned in the course to students' home schools, which then assign a letter grade based on the local grading 
scale, so benchmarks for passing grades can vary across schools or districts. For Wisconsin Virtual School, 
students must complete 95-100 percent of the possible course activities to obtain a certificate of completion. 
The home district then determines whether 95-100 percent course activity completion with a passing grade, 
which is determined locally, provides an earned credit or advancement. 

Source: Authors' analysis of Wisconsin Virtual School data (fall 2014 semester). 



Students who engaged in their online course for two or more hours per week had better outcomes 
than students who engaged for fewer than two hours per week 

After student and course characteristics were controlled for, students who engaged for 2 
hours or more per week (the initial 2 hours with spike, steady 2.5 hours, steady 4+ hours, 
and variable 6+ hours groups) had better course outcomes than students who engaged con- 
sistently during the semester with less time each week (the steady 1.5 hours group). 

Of particular interest is the comparison between the steady 1.5 hours and steady 2.5 hours 
groups, which account for a large majority of the sample and had similar engagement pat- 
terns, with the exception that the steady 2.5 hours group was logged in an average of 
approximately one additional hour per week. The average percentage of possible points 
earned in the course was 10 percentage points higher for the steady 2.5 hours a week 
engagement group than for the steady 1.5 hours group, and the average percentage of course 
activities completed was 7 percentage points higher for the steady 2.5 hours group than for 
the steady 1.5 hours group (see table 2). The differences were statistically significant. 

Despite their unique engagement patterns, students who engaged in their online course for two or 
more hours per week had similar course outcomes 

Four engagement groups (initial 2 hours with spike, steady 2.5 hours, steady 4+ hours, and 
variable 6+ hours) had similar course outcomes on average (see table 2). This indicates 


The average 
percentage of 
possible points 
earned in the 
course was 
10 percentage 
points higher for 
the steady 2.5 
hours a week 
engagement group 
than for the steady 
1.5 hours group, 
and the average 
percentage of 
course activities 
completed was 
7 percentage 
points higher for 
the steady 2.5 
hours group than 
for the steady 
1.5 hours group 
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Table 2. Students who engaged for two or more hours per week outperformed 
students who engaged for fewer than two hours per week, and students who 
engaged 1.5 hours per week steadily across the semester outperformed students 
whose engagement decreased over the semester, fall 2014 



Percentage of possible points 
earned in the course 

Percentage of course 
activities completed 

Engagement group 

Mean 3 

Standard 

deviation 

Mean 3 

Standard 

deviation 

Initial 1.5 hours with decrease 

4^_*** 

35 


27 

Steady 1.5 hours 

76 

21 

91 

18 

Initial 2 hours with spike 

85*** 

14 

99** 

10 

Steady 2.5 hours 

gQ* ** 

13 

gg*** 

7 

Steady 4+ hours 

88** 

14 

100** 

10 

Variable 6+ hours 

89* 

8 

100* 

0.5 


* is statistically significantly different from the mean for the steady 1.5 hours group at p < .05; ** is sta- 
tistically significantly different from the mean for the steady 1.5 hours group at p < .01; *** is statistically 
significantly different from the mean for the steady 1.5 hours group at p < .001. 

a. Statistically adjusted for student demographic characteristics and course type taken (Advanced Placement, 
core, or elective). See table B3 in appendix B for regression coefficients. 

Source: Authors' analysis of Wisconsin Virtual School learning management system data (fall 2014 semester). 


that students (all of whom were allowed to engage in their online course at their own pace) 
were likely to succeed in terms of the percentage of possible points earned in the course 
and the percentage of possible course activities completed as long as they were logged in to 
their online course for at least two hours per week on average. 

Implications of the study findings and directions for future research 


This study provides state policymakers, state and local education agencies, and online 
learning providers with information about student engagement in online courses and how 
patterns of engagement are associated with course outcomes. 

Because the findings are based on correlational data, they cannot be used to make causal 
connections between student engagement and online course outcomes. Although patterns 
of high engagement appear to be associated with better course outcomes, it is possible that 
a third unmeasured variable is associated with course outcomes beyond student engage- 
ment (such as student interest in content, motivation, or access to technology). 

Wisconsin Virtual School directors and directors of other online learning programs may 
be able to use information from this study as they seek ways to support student success 
in online courses, especially students in the initial 1.5 hours with decrease and steady 
1.5 hours engagement groups, which had poorer course outcomes. Supports might include 
having online teachers and onsite facilitators target students who log in for fewer than two 
hours per week to see whether they are struggling with course content or need assistance 
with skills such as time management. Online teachers might also consider contacting the 
student, parents, or the brick-and-mortar school to provide notice that the student is at 
risk of failing the course based on his or her limited participation. 


Although 
patterns of high 
engagement 
appear to be 
associated with 
better course 
outcomes, it is 
possible that a 
third unmeasured 
variable is 
associated with 
course outcomes 
beyond student 
engagement 
(such as student 
interest in content, 
motivation, 
or access to 
technology) 


Other online learning programs across the country may be able to use the findings as a 
framework for investigating the data available in their learning management systems and 
student information systems. In combination with future studies investigating how much 
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these findings can be generalized to students in other online learning programs, the results 
may be able to inform the provision of support to students who demonstrate engagement 
patterns that are associated with poor course outcomes. 


In addition to informing policies and practices for supporting students in online courses, the 
current study could inform a future comprehensive study investigating associations among 
student engagement, online teacher and onsite facilitator support, and online course out- 
comes. Researchers could collect data related to teacher and facilitator support by, for example, 
administering a survey about support practices or encouraging educators to communicate with 
students using the messaging capabilities in the learning management system so that commu- 
nications are stored for later analysis. 2 Such a study might confirm the relationship between 
student engagement and course outcomes found by the current study and extend the model 
to account for teachers’ and facilitators’ support. Future research might also explore whether 
certain teaching activities or course requirements lead to different engagement patterns. 

Limitations of the study 


This study has three main limitations. 

First, the findings may not extend to other types of online courses or other online learn- 
ing programs. The study sampled student enrollments in Advanced Placement courses, core 
courses, and elective courses because these courses represent the majority of Wisconsin 
Virtual School enrollments. The findings are not generalizable to credit recovery courses 
because they were excluded from the sampling plan due to their shortened, 12 week timeline. 
It also is possible that the findings do not represent the engagement patterns or relationships 
between engagement and course outcomes evident in other online learning programs. 

Second, the amount of time a student is logged in to a learning management system is an 
imperfect measure of student engagement. Students are not necessarily actively working 
while logged in, and some courses may require engagement in academic activities outside the 
system, such as writing reports or reading books. However, the learning management system 
logs students out after five minutes of inactivity, which should reduce some of the error asso- 
ciated with these measures. The study team was unable to capture other measures of engage- 
ment such as communications among students, onsite facilitators, and online teachers, as 
nearly all course communications occur outside the learning management system, through 
school email or phone calls, and are therefore not captured in any existing datasets. 


In addition to 
informing policies 
and practices 
for supporting 
students in online 
courses, the 
current study could 
inform a future, 
comprehensive 
study investigating 
associations 
among student 
engagement, 
online teacher and 
onsite facilitator 
support, and online 
course outcomes 


Third, the analysis methods did not fully account for the complex structure of online course 
data. It is possible that students with the same facilitator, school, or online teacher have 
more similar outcomes than students who do not share these relationships. The analyses 
accounted for the relationships among students only within online course sections, which 
does not capture all possible relationships in the data. As typically implemented, online 
courses involve a complex structure, with students assigned to particular onsite facilitators, 
brick-and-mortar schools, online courses, and online teachers. Further complications arise 
when considering that students in the same brick-and-mortar school might be enrolled in 
sections of the same course with different online teachers or in different courses with the 
same online teacher. Because the analysis did not capture all these possible relationships, it 
is possible that the results do not accurately estimate the associations between engagement 
group membership and course outcomes. 
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Appendix A. Study methodology 


This appendix describes the sampling strategy; data sources, instruments, and collection 
methods; and data processing and analysis in general and for each research question. 

Sampling strategy 

The sample included all Wisconsin Virtual School high school course enrollments from 
the fall 2014 semester for which the student logged in to the learning management system 
for the first time between August 1 and September 30, 2014- This sampling strategy 
excluded enrollments in credit recovery courses, because the duration of those courses 
is shorter than Advanced Placement, core, and elective courses. 3 Students who dropped 
their course during Wisconsin Virtual School’s 14 day grace period were also excluded. 
The study sample included 1,512 student enrollments in 109 courses (19 Advanced Place- 
ment courses, 44 core courses, and 46 elective courses). Students could enroll in more 
than one online course, so the sample represents 1,179 unique students, 170 of whom took 
more than one online course during the fall 2014 semester. Each student enrollment in an 
online course represented a single “case” in the analysis. 

Because Wisconsin Virtual School uses a rolling enrollment system, students could have 
enrolled in a course at any time and could have had up to five months to complete a semes- 
ter course. Wisconsin Virtual School directors reported that most students taking courses 
during fall 2014 would have enrolled by the end of September and completed their course 
by the end of February 2015. As a result, the study used data for students who enrolled in 
the sampled courses between August 1 and September 30, 2014, including students who 
completed the course and students who dropped the course any time prior to February 28, 
2015. Any student who enrolled in the course after September 30 was not included in the 
sample because his or her outcome data were not available in time to be included in the 
current study, and the analysis approach required that all included students have course 
outcome data. 

Data sources, instruments, and collection methods 

The research questions were addressed using data from Wisconsin Virtual School’s learn- 
ing management system and student information system. The data included the following 
elements: students’ online course activity (the date, time, and duration of logins in hours), 
students’ demographic characteristics, online course characteristics, and students’ scores 
on quizzes and exams. 

General data processing and analysis 

Prior to conducting the planned analyses, several steps were taken to prepare the data for 
analysis. First, the learning management system reports zero values when students do not 
log in during a particular week, regardless of their status in the course, and some of the 
values needed to be deleted. The raw data contained three types of zero values: zeros indi- 
cating that a student did not log in to the learning management system during a particular 
week but logged in during subsequent weeks, zeros indicating that a student was no longer 
attempting to complete the course but had not formally dropped the course, and zeros indi- 
cating that a student was not logging in because he or she had successfully completed the 
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course. An enrollment status variable was used to determine which students were actively 
enrolled and which had successfully completed the course. For students who had not com- 
pleted the course but were still attempting to do so and students who did not intend to 
complete the course but had not formally dropped the course, all values were retained, as 
the values represent cases in which the student could still log in to the course. For students 
who completed the course, all zeros after the last nonzero login value were deleted, because 
a student who had successfully completed the course was not expected to log in. 

Because the analysis approach described later is sensitive to spikes or fluctuations in values 
across repeated measurements such that the model would not converge using the raw data, 
the data were top-coded at the global 99th percentile of 14 hours per week (all values 
greater than 14 were replaced with 14). This procedure should not influence the results 
because these extreme values were rarely observed (only 1 percent of values were replaced), 
and top-coding allows all available cases to be included in the analysis. 

Finally, the time variable was recoded from weeks in the online course (1-21) to percent- 
age of the course (week/21). The data were recoded in this way to reduce the range and 
avoid the propagation of extreme values in the estimation process. 

Analysis for the research question on student engagement patterns 

To answer the first research question (How many student engagement patterns are evident 
in online courses, and what percentage of student enrollments follow each pattern?), the 
study team used a group-based trajectory modeling approach (Nagin, 2005) to examine 
patterns of student engagement across the duration of the online course and to classify 
the patterns into unique groups. Group-based trajectory modeling is an extension of tra- 
ditional growth curve modeling; both model individual-level variation in developmental 
trajectories. Group-based trajectory modeling differs from standard growth curve mod- 
eling because group-based trajectory modeling assumes that there may be an unknown 
distribution within the population of individual differences within groups, making it a 
nonparametric method for examining growth, rather than assuming that the distribution 
of trajectories in the population follows a multivariate normal distribution (Nagin, 2005). 
As output from the group-based trajectory modeling analyses, each case receives an esti- 
mate of assignment probability to each latent group and can subsequently be classified into 
qualitatively distinct trajectory groups. This approach is particularly useful for applications 
such as the current study, in which there is no theoretical basis for assuming qualitatively 
similar patterns of change across individuals (Andruff, Carraro, Thompson, Gaudreau, & 
Louvet, 2009); in other words, engagement might increase for some students and decrease 
for others across the duration of the course. 

Student engagement was measured by the amount of time each student was logged in to 
the learning management system. Weekly aggregate measures were created by summing 
the login amounts for each week (Monday through Sunday), beginning with the week the 
student first logged in to the learning management system and ending when the student 
completed the course or the five-month course period elapsed. Students therefore had dif- 
ferent numbers of engagement scores depending on the rate at which they finished the 
course. Group-based trajectory modeling allows for different timing and frequencies of 
measurement occasions across cases (Nagin, 2005), so this approach naturally accommo- 
dates this issue. Because students were given a maximum of five months to complete their 
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course, approximately 21 data points were available for each student. Students who com- 
pleted their course more quickly had fewer than 21 data points. 

After the data were prepared for analysis, a group-based trajectory modeling procedure was 
conducted using STATA. After loading a trajectory estimation software package, 4 three 
general steps of statistical modeling were conducted: model selection, model estimation, 
and model interpretation. 

Step 1: Model selection. Selecting a group -based trajectory model involves two stages: 
model enumeration and model specification. Model enumeration involves determining 
the number of latent groups with unique growth trajectories. Model specification involves 
parameterizing the multigroup models by including the order of the growth trajectories 
(such as linear, quadratic, or cubic forms) for each group. First, a baseline, single-group tra- 
jectory model was generated using a censored normal (Tobit) model. 5 Subsequent models 
were then fit with increasing numbers of latent growth engagement groups (in other words, 
a two-group solution was fit, followed by a three-group solution, then a four-group solution, 
and so on). 

To specify each model’s optimal solution, higher order slope terms were assessed for each 
group’s trajectory by fitting a series of models. First, a fully specified model with quintic 
trajectories for each group was fit. In the next run the highest order nonsignificant param- 
eter for each group was trimmed. After parameters were trimmed, the more parsimoni- 
ous model was re-run. Modeling followed this iterative trimming process until each of the 
remaining highest order parameters achieved significance for each group. This trimming 
process resulted in an estimation of the most parsimonious solutions for two groups, three 
groups, four groups, five groups, and six groups. Modeling was ceased at six groups because 
additional groups resulted in redundant trajectory groups. 

As latent groups are added, the Bayesian information criterion estimates (Schwarz, 1978) 
change. By examining the two Bayesian information criterion statistics, one using the 
number of student enrollments to estimate degrees of freedom (n = 1,512) and another 
using the number of student engagement data points (n = 29,109), the best-fitting model 
can be selected using guidelines outlined in Andruff et al. (2009), Kreuter and Muthen 
(2008), and Nagin (1999, 2005), with higher Bayesian information criterion values (closer 
to zero) indicating better model fit. Bayesian information criterion fit statistics favor par- 
simonious models, ensuring that the increase in explanatory power that results from the 
complexity associated with adding parameters is statistically warranted. Although the 
Bayesian information criterion necessarily gets larger as parameters are added, the fit statis- 
tic also assesses a penalty for adding to the model’s complexity. The goal of this process is 
to identify distinctive features in the data that result in a set of groups with unique engage- 
ment patterns; both Bayesian information criterion statistics support a six-group solution 
for the trajectory analysis (table Al). 

Step 2: Model estimation. After identifying the number of unique student engagement 
trajectory groups and the best-fitting trajectory forms, the study team estimated the result- 
ing six group trajectory parameters in STATA, obtaining posterior probabilities of group 
membership for each student enrollment along with each student’s estimated group assign- 
ment using the maximum probability assignment rule. Among students who were assigned 
to each group, the average of the estimated probability of membership in that group was 
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Table Al. Fit statistics for model enumeration process, fall 2014 


Bayesian information criterion fit statistics 

Model 

With number of student 
enrollments used to estimate 
degrees of freedom 
(n = 1,512) 

With number of student 
engagement data points used to 
estimate degrees of freedom 
(n = 29,109) 

Two group 

-62,357 

-62,373 

Three group 

-61,376 

-61,404 

Four group 

-60,918 

-60,948 

Five group 

-60,551 

-60,591 

Six group 

-60,338 

-60,384 


Note: Larger values (closer to zero) indicate better model fits. 

Source: Authors' analysis of Wisconsin Virtual School data (fail 2014 semester). 


calculated to provide a measure of internal reliability for each group, with all average prob- 
abilities greater than .70 indicating acceptable reliability (Andruff et al., 2009; average 
predicted probabilities of group membership are provided in table A2). 

Step 3: Model interpretation. A figure was created to depict the estimated shapes of the 
resulting group patterns throughout the duration of the online course for each group. The 
engagement trajectories were stylized by connecting the predicted values for each week. 
Groups were named based on the resulting average weekly login durations and variability 
in login durations across the semester. 

Analysis for the research question on differences in student engagement patterns by course and 
student characteristics 

To answer the second research question (Do student engagement groups differ in course 
type taken, gender, or grade level?), group profiles were assembled for each engagement 
group. Indicators of group membership were created using the group assignment probabih 
ities for each student determined through the group-based trajectory modeling analysis. 


Table A2. Average group assignment probabilities for student engagement groups, 
fall 2014 

Assigned group 

Number 

assigned 

1 

Model identified group 

2 3 4 5 

6 

Range 3 

Initial 1.5 hours with decrease (group 1) 

122 

.961 

.038 

.000 

.001 

.000 

.000 

.50-1.00 

Steady 1.5 hours (group 2) 

592 

.007 

.890 

.011 

.011 

.000 

.000 

.44-1.00 

Initial 2 hours with spike (group 3) 

68 

.000 

.043 

.914 

.003 

.000 

.000 

.47-1.00 

Steady 2.5 hours (group 4) 

572 

.000 

.010 

.015 

.859 

.026 

.000 

.42-1.00 

Steady 4+ hours (group 5) 

122 

.000 

.000 

.004 

.004 

.915 

.008 

.42-1.00 

Variable 6+ hours (group 6) 

36 

.080 

.000 

.000 

.000 

.046 

.953 

.52-1.00 


Note: The model assigns each student a probability that he or she belongs in each group. Under perfect condi- 
tions the probability of actual group assignment would be 1 for one group and 0 for all others. The model-identified 
group column heads refer to the six groups identified by the model. The assigned group row labels represent the 
groups to which students were actually assigned based on the maximum probability rule. The estimates in the 
table represent the average model assignment probabilities across students in each assigned group. 

a. Reports the highest and lowest model assignment probabilities across students for the group to which they 
were assigned. 

Source: Authors' analysis of Wisconsin Virtual School data (fall 2014 semester). 


A-4 






Descriptive statistics were calculated, including means, standard deviations, and percent- 
ages for each group. A series of chi-squared analyses were conducted to determine whether 
differences across groups in course type taken, gender, and grade level were statistically 
significant. 

Analysis for the research question on associations between student engagement patterns and 
course outcomes 


To answer the third research question (How are student engagement patterns associated 
with the percentage of possible points earned in the course and the percentage of course 
activities completed?), the study team statistically assessed the relationships between 
engagement group membership and performance in the online course while controlling for 
course type taken, student gender, and grade level. The study team conducted a series of 
two-level regression analyses with students nested within online courses using hierarchical 
linear modeling (Raudenbush & Bryk, 2002). 


The two measures of student learning outcomes, percentage of possible points earned in 
the course and percentage of course activities completed, were included as dependent vari- 
ables in two separate models (y fj ). Five dummy-coded indicators of engagement group mem- 
bership were included as independent variables (ENG mj .). To examine engagement group 
effects, it was necessary to select a reference group for comparison. The steady 1.5 hours 
engagement group was chosen because it was the largest group and because the average 
percentage of possible points earned in the course and percentage of course activities com- 
pleted were closest to the whole sample’s averages. Gender and grade level were included 
as covariates (ST ..), with gender dummy-coded such that zero values were male students 
and values of 1 were female students. Grade level was grand mean-centered at grade 11. 
Since the majority of enrollments were in grades 10, 11, and 12, this approach allowed 
the study team to estimate the intercept for a grade that is at the center of the distri- 
bution of grade levels. Online course-type variables, elective and Advanced Placement, 
were added as dummy-coded indicators at level 2 to control for potential differences in 
the outcomes based on course type taken (OC^.), with core courses as the reference group. 
Cross-level interactions were included for online course type taken and engagement group 
(ENG mi *OC fj ). All effects were modeled as fixed. The general form of the model was as 
follows: 


r. 

•nj 


: Too + o ENG nuj + 'Zrqo ST q ,j + i>0 f oc fj + £ iy r 

m=l q=6 f= 1 m=l f=l 


rENG .. * OCr. + u r . 

if ml] f] Oj 


+ r... 
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Appendix B. Supplemental findings 


This appendix provides supplemental statistical results for each research question. 


Table Bl. Relationships between course type taken, gender, and grade level and 
course outcomes, fall 2014 


Characteristic 

Number 

Percentage of possible 
points earned in the course 

Percentage of course 
activities completed 

of student 
enrollments 

Mean 

Standard 

deviation 

Mean 

Standard 

deviation 

Course type taken 

Advanced Placement 

229 

81.45 

19.37 

93.05 

19.15 

Core 

539 

76.61 

23.19 

87.92 

25.97 

Elective 

744 

80.06 

21.54 

90.13 

23.38 

Gender 3 

Female 

897 

79.75 

21.76 

90.36 

23.14 

Male 

604 

77.73 

22.18 

88.75 

24.95 

Grade level 3 

5-8 

43 

76.92 

16.67 

90.51 

22.36 

9 

170 

76.38 

24.22 

84.98 

27.52 

10 

381 

79.49 

21.67 

90.36 

23.24 

11 

421 

78.98 

20.60 

91.05 

21.15 

12 

486 

79.54 

22.84 

89.62 

25.25 


a. 11 student enrollments (less than 1 percent of the total) were missing information on gender or grade level. 
Percentages exclude missing values. 

Source: Authors' analysis of Wisconsin Virtual School data (fall 2014 semester). 
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Table B2. Relationships between course type taken, gender, and grade level and 
engagement groups, fall 2014 (chi-squared statistic) 

Characteristic and 
engagement group 

Steady 1.5 
hours 

Initial 2 
hours with 
spike 

Steady 2.5 
hours 

Steady 4+ 
hours 

Variable 

6+ hours 

Course type taken (Advanced Placement, core, 

elective) 3 




Initial 1.5 hours with decrease 

8.09* 

13.44** 

12.79** 

24.49*** 

10.39** 

Steady 1.5 hours 


18.79*** 

28.50*** 

49.24*** 

15.00** 

Initial 2 hours with spike 



1.69 

0.83 

1.42 

Steady 2.5 hours 




9.98** 

3.36 

Steady 4+ hours 





1.91 

Gender (male, female) 11 

Initial 1.5 hours with decrease 

0.64 

3.80 

0.38 

0.15 

0.78 

Steady 1.5 hours 


8.75** 

0.09 

0.09 

0.28 

Initial 2 hours with spike 



7.87** 

5.19* 

4.92* 

Steady 2.5 hours 




0.01 

0.39 

Steady 4+ hours 





0.40 

Grade (5-8, 9, 10, 11, 12) c 

Initial 1.5 hours with decrease 

17.52** 

12.98* 

16.56** 

8.63 

9.45 

Steady 1.5 hours 


2.63 

11.04 

4.38 

9.94 

Initial 2 hours with spike 



5.07 

5.99 

3.99 

Steady 2.5 hours 




8.43 

4.32 

Steady 4+ hours 





8.42 


* is significant at p < .05; ** is significant at p < .01; *** is significant at p < .001. 

a. Assessed using chi-squared ratio with two degrees of freedom. 

b. Assessed using chi-squared ratio with one degree of freedom. 

c. Assessed using chi-squared ratio with four degrees of freedom. 

Note: p-values are not adjusted for multiple comparisons. 

Source: Authors' analysis of Wisconsin Virtual School data (fall 2014 semester). 
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Figure Bl. Engagement patterns with confidence bands, fall 2014 


Hours per week spent logged in to the online course 


12 

10 

8 

6 

4 

2 

0 



Steady 1.5 hours (39 percent) 


Initial 2 hours with spike (4 percent) 


Variable 6+ hours (2 percent) 


Steady 4+ hours (8 percent) 


iniiial 1.5 hours with decrease (8 percent) 
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Weeks in online course 


Note: Analysis is based on 1,512 student enrollments. The maximum number of data points for each student 
enrollment is 21 weeks, though some students finished their course in fewer than 21 weeks. This approach 
uses ail available data for each week. Values in parentheses are the percentage of the sample that followed 
each engagement pattern; percentages do not sum to 100 because of rounding. The student engagement pat- 
terns depicted were stylized using the predicted values for each week. The model also estimates population 
percentages, which were very similar to the sample percentages: initial 1.5 hours with decrease, 8 percent; 
steady 1.5 hours, 39 percent; initial 2 hours with spike, 5 percent; steady 2.5 hours, 37 percent; steady 
4+ hours, 9 percent; and variable 6+ hours, 2 percent. Dashed lines depict 95 percent confidence intervals 
around the estimated trajectories. 

Source: Authors' analysis of Wisconsin Virtual School data (fall 2014 semester). 
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Table B3. Regression coefficients for course outcome analyses, fall 2014 
(percentage points) 

Fixed effect 

Percentage of possible points 
earned in the course 

Percentage of course 
activities completed 

Intercept 

72.52*** 

89.65*** 

Elective 

5.39* 

1.85 

Advanced Placement 

-3.73 

-3.62 

Gender 

1.38 

0.68 

Grade level 

-0.04 

-0.01 

Initial 1.5 hours with decrease 

-29.98*** 

-62.62*** 

Elective 

0.08 

4.64 

Advanced Placement 

1.10 

1.99 

Initial 2 hours with spike 

13.73*** 

10.29** 

Elective 

-7.88 

-4.51 

Advanced Placement 

2.08 

1.99 

Steady 2.5 hours 

9.91*** 

7.28*** 

Elective 

-2.30 

-1.88 

Advanced Placement 

8.61* 

5.48 

Steady 4+ hours 

9.91** 

7.76** 

Elective 

1.76 

2.03 

Advanced Placement 

10.19 

5.21 

Variable 6+ hours 

10.27* 

10.18* 

Elective 

6.93 

1.34 

Advanced Placement 

10.72 

3.45 


* is significant at p < .05; ** is significant at p < .01; *** is significant at p < .001 

Note: The steady 1.5 hours group was the reference group for engagement group, and core course was the ref- 
erence group for course type taken. The intercept coefficient represents the score estimate for the steady 1.5 
hours group for male students in core courses. The coefficients next to each group represent the difference in 
its estimate compared with the reference group (for example, the initial 1.5 hours with decrease group aver- 
ages 29.98 percentage points less than the steady 1.5 hours group). The coefficients that appear underneath 
each group (elective, Advanced Placement, gender, and grade level) represent the conditional effects of being 
in an elective course, a female student, or in a different grade level). The grade level was centered at grade 
11 . 

Source: Authors' analysis of Wisconsin Virtual School data (fall 2014 semester). 
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Notes 


1. Analyses focus on the total amount of time spent logged in each week, but the fre- 
quency of logins each week can vary within engagement groups. For example, for two 
students both logging in a total of two hours in a particular week, one student may 
have logged in one time for two hours, while the other student may have logged in 
four times for 30 minutes each. 

2. A review of Wisconsin Virtual School’s learning management system data revealed 
that most communications among teachers, facilitators, and students occur through 
school email rather than through the learning management system, which means that 
those communications were not accessible for analysis. 

3. Credit recovery courses must be completed in 12 weeks, while all other courses must 
be completed in five months, which would result in necessarily different engagement 
trajectories. 

4. Trajectory estimation software was downloaded from http://www.andrew.cmu.edu/user/ 
bjones/index.htm. 

5. The STATA Traj package does not include an option for a standard normal, but the 
censored minimums (0 hours) and maximums (14 hours) for the current analyses were 
set such that they encompassed all the data. This model is therefore equivalent to a 
standard normal model. 
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